verification objective
TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generation
Nadimi, Bardia, Filom, Khashayar, Chen, Deming, Zheng, Hao
With the rapid advancement of Large Language Models (LLMs), there is growing interest in applying them to hardware design and verification. Among these stages, design verification remains the most time-consuming and resource-intensive phase, where generating effective stimuli for the design under test (DUT) is both critical and labor-intensive. We present {\it TB or not TB}, a framework for automated stimulus generation using LLMs fine-tuned through Coverage-Driven Direct Preference Optimization (CD-DPO). To enable preference-based training, we introduce PairaNet, a dataset derived from PyraNet that pairs high- and low-quality testbenches labeled using simulation-derived coverage metrics. The proposed CD-DPO method integrates quantitative coverage feedback directly into the optimization objective, guiding the model toward generating stimuli that maximize verification coverage. Experiments on the CVDP CID12 benchmark show that {\it TB or not TB} outperforms both open-source and commercial baselines, achieving up to 77.27\% improvement in code coverage, demonstrating the effectiveness of Coverage-driven preference optimization for LLM-based hardware verification.
Export Reviews, Discussions, Author Feedback and Meta-Reviews
First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. This paper's core finding is that combining an identity classification task as well as metric-learning-style verification task helps to learn better features for face classification/verification. The verification task here tries to decrease feature-space distance between instances of the same identity, and increase distance between those of different identities. This improvement is embedded in a state-of-the-art system for face verification, which uses convnets trained on many (400) different views to generate features, distilled into a small set of 25 using feature selection. Very good results are obtained and experiments performed using LFW as a test set. Overall, these are very good results obtained using a somewhat complex pipeline, and a good investigation into the contribution of each task in the loss for feature learning.
Toward Certification of Machine-Learning Systems for Low Criticality Airborne Applications
Dmitriev, K., Schumann, J., Holzapfel, F.
The exceptional progress in the field of machine learning (ML) in recent years has attracted a lot of interest in using this technology in aviation. Possible airborne applications of ML include safety-critical functions, which must be developed in compliance with rigorous certification standards of the aviation industry. Current certification standards for the aviation industry were developed prior to the ML renaissance without taking specifics of ML technology into account. There are some fundamental incompatibilities between traditional design assurance approaches and certain aspects of ML-based systems. In this paper, we analyze the current airborne certification standards and show that all objectives of the standards can be achieved for a low-criticality ML-based system if certain assumptions about ML development workflow are applied.
A Dual Approach to Scalable Verification of Deep Networks
Dvijotham, Krishnamurthy, Stanforth, Robert, Gowal, Sven, Mann, Timothy, Kohli, Pushmeet
This paper addresses the problem of formally verifying desirable properties of neural networks, i.e., obtaining provable guarantees that the outputs of the neural network will always behave in a certain way for a given class of inputs. Most previous work on this topic was limited in its applicability by the size of the network, network architecture and the complexity of properties to be verified. In contrast, our framework applies to much more general class of activation functions and specifications on neural network inputs and outputs. We formulate verification as an optimization problem and solve a Lagrangian relaxation of the optimization problem to obtain an upper bound on the verification objective. Our approach is anytime, i.e. it can be stopped at any time and a valid bound on the objective can be obtained. We develop specialized verification algorithms with provable tightness guarantees under special assumptions and demonstrate the practical significance of our general verification approach on a variety of verification tasks.